Holdout-Based Empirical Assessment of Mixed-Type Synthetic Data

AI-based data synthesis has seen rapid progress over the last several years and is increasingly recognized for its promise to enable privacy-respecting high-fidelity sharing. This reflected by growing availability of both commercial open-sourced software solutions synthesizing private data. However, despite these recent advances, adequately evaluating quality generated synthetic datasets still an open challenge. We aim close this gap introduce a novel holdout-based empirical assessment framework quantifying fidelity as well privacy risk mixed-type tabular Measuring based on statistical distances lower-dimensional marginal distributions, which provide model-free easy-to-communicate metric representativeness dataset. Privacy assessed calculating individual-level closest record with respect training By showing that samples are just holdout data, we yield strong evidence synthesizer indeed learned generalize patterns independent individual records. empirically demonstrate presented seven distinct across four compare then traditional perturbation techniques. Both Python-based implementation proposed metrics demonstration study setup made available open-source. The results highlight need systematically assess emerging class generators.